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FOREWORD 


This  research  and  development  was  conducted  within  exploratory  development  task 
ZF63-522-001-013-03.02  (Computer-based  Techniques  for  Training  Tactical  Knowledge). 
It  was  sponsored  by  the  Office  of  the  Chief  of  Naval  Operations  (OP-01)  with  support 
from  the  Defense  Advanced  Research  Projects  Agency  (CTO). 

This  report  describes  the  development  and  evaluation  of  a  computer-based  memoriza¬ 
tion  system  for  training  tactical  action  officers  (TAOs).  Ultimately,  this  system  can  be 
used  within  TAO  training  to  evolve  student  models,  computer  tutors,  and  gaming 
techniques  to  facilitate  the  learning  of  declarative  information.  Details  of  some  of  the 
earlier  work  on  this  effort  were  documented  in  a  previous  report  (NPRDC  TN  81-8). 

Results  of  this  R&D  are  intended  primarily  for  designers  of  computer-based 
instruction  and  the  operational  tactical  training  community. 


JAMES  F.  KELLY,  JR.  JAMES  W.  TWEEDDALE 

Commanding  Officer  Technical  Director 
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SUMMARY 


Problem 


Increasing  technological  sophistication  and  continuing  scarcities  in  the  supply  of 
qualified  personnel  have  created  a  need  for  improved  Navy  training,  particularly  in  the 
area  of  critical  combat  skills.  Emerging  computer  technologies  may  offer  potential 
benefits  for  Navy  training.  Although  the  military  has  been  performing  research  in 
computer-assisted  instruction  (CAI)  since  1968,  very  little  work  has  been  done  using 
microcomputers.  This  is  also  true  in  regard  to  evaluating  "intelligent"  CAI  systems  for 
military  training,  even  though  several  ideas  associated  with  artificial  intelligence  (AI) 
offer  promise  for  improved  training.  Microcomputer  technologies,  as  well  as  recent 
developments  in  AI,  need  to  be  evaluated  in  applied  settings. 

Purpose 

The  research  described  herein  is  part  of  a  3-year  developmental  effort.  The  overall 
purpose  is  to  evaluate  microprocessors  for  Navy  training  in  a  context  that  permits  the 
exploration  of  training  techniques  related  to  AI  and  cognitive  science.  The  specific 
objectives  are:  (1)  to  create  a  usable  tactical  memorization  training  system,  (2)  to  gain 
experience  in  the  in  situ  development  of  stand-alone  CAI  systems  for  Navy  training,  (3)  to 
explore  gaming  techniques  to  facilitate  the  acquisition  of  declarative  information,  and  (4) 
to  use  this  developmental  situation  to  evolve  student  models  and  computer  tutors. 

Approach 

Tactical  action  officer  (TAO)  training  was  selected  as  the  domain  for  this  research 
because  a  major  segment  of  this  training  involves  memorization  of  declarative  informa¬ 
tion.  Further,  the  Navy  has  a  need  to  facilitate  the  acquisition  and  retention  of  this  type 
of  information. 

A  data  base  of  facts  involving  Soviet  platforms  and  weapons  systems  was  compiled, 
represented  as  a  semantic  network,  and  implemented  on  a  TERAK  microcomputer.  This 
microcomputer  was  installed  at  the  Fleet  Combat  Training  Center,  Pacific  (FCTCP) 
where  TAO  training  is  conducted.  A  series  of  games  that  permit  drill  and  practice  on 
information  in  the  data  base  and  audit  trails  that  collect  extensive  information  on  student 
use  of  the  game  were  developed.  The  primary  advantage  of  this  type  of  development  is 
that  the  gaming  software  may  be  considered  a  separate  entitity  from  the  data  base. 
These  games  can  be  run  on  any  data  base  that  can  be  represented  as  a  semantic  network. 
Three  other  data  bases  were  developed  to  test  this  claim. 

All  developmental  work  was  carried  out  in  a  school  setting.  There  was  extensive 
participation  on  the  part  of  the  students  and  instructors  at  FCTCP  throughout  the  project. 
Ideas  were  discussed  with  the  users  before  work  was  started,  their  input  was  used  in 
compiling  the  data  base,  and  one  microprocessor  was  installed  at  the  school  with  the  very 
earliest  on-line  development.  Informal  feedback  solicited  from  school  personnel  in  the 
early  stages  resulted  in  many  changes  in  the  games. 

Results 


At  this  time,  systems  are  installed  at  FCTCP  and  students  are  using  the  games  on  a 
regular  basis.  The  games  are  designed  to  supplement  the  TAO  course  and  students  may 
use  them  or  other  study  methods. 
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The  games  have  been  available  to  students  from  three  TAO  classes  (N  =  54).  About 
half  of  these  students  have  used  the  games  and  data  have  been  collected  primarily  for 
formative  evaluation.  Measures  of  school-administered  tests,  computer  audit  trails,  and 
attitude  questionnaires  were  collected. 

A  number  of  changes  were  made  in  the  interface  for  the  system  as  a  result  of 
informal  feedback  and  the  questionnaire  data.  It  became  clear  over  time  that  the 
structure  of  the  interface  is  a  strong  determinant  of  how  much  time  students  are  willing 
to  spend  on  the  system.  Although  not  all  students  chose  to  use  the  gaming  system,  those 
who  did  had  positive  feelings  about  it  and  preferred  some  games  more  than  others.  The 
instructors  were  enthusiastic  and  supportive  of  the  research. 

Conclusions 


The  gaming  system  developed  can  be  used  for  tactical  training  and  offers  promise  for 
additional  development  planned  for  the  last  year  of  the  effort.  In  situ  development 
facilitated  the  effort,  provided  new  directions,  and  ensured  user  support.  The  work  was 
generalized  to  other  training  domains  with  the  development  of  additional  data  bases. 
Additionally,  the  flexibility  of  microprocessors  for  this  kind  of  Navy  training  was 
demonstrated. 

Future  Direction 


The  gaming  system  is  being  transitioned  into  advanced  development  funding  as  a  part 
of  the  Computer-based  Educational  Support  System  project. 
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Problem 


INTRODUCTION 


There  is  a  continuing  need  for  improved  Navy  training,  particularly  for  critical 
combat  skills.  As  the  technologies  supporting  weapons  systems  have  become  more 
sophisticated,  the  Navy  has  experienced  a  decline  in  trained  personnel.  Means  are  needed 
to  train  people  quickly,  effectively,  and  at  low  cost.  Emerging  technologies  in 
combination  with  new  and  existing  training  methodologies  need  to  be  investigated  in 
response  to  the  training  problem. 

New  developments  in  computer  technologies,  particularly  microcomputer-based 
systems,  offer  potential  benefits  for  training.  These  systems  represent  increased 
computational  power  in  a  smaller,  portable  configuration  at  lower  cost.  There  is  a  need 
to  evaluate  such  systems  in  applied  settings  to  determine  their  advantages  and  dis¬ 
advantages  for  computer-assisted  instruction  (CAI)  in  Navy  training. 

There  is  also  a  need  to  explore  the  applicability  of  training  methodologies  related  to 
cognitive  science  and  artificial  intelligence  (AI)  for  Navy  computer-based  systems. 
Intelligent  CAI  may  offer  the  increased  individualization  required  for  the  wide  range  of 
abilities  people  have  in  initial  training  and  the  automated  tutor  capabilities  needed  for 
refresher  training  outside  of  schools. 

Background 

The  military  has  a  history  of  interest  in  computer  applications  in  training.  Since 
1968,  researchers  in  all  of  the  services  have  collected  data  from  a  wide  variety  of 
experimental  programs.  Their  interests  have  focused  on  evaluations  of  particular  CAI 
systems  (e.g.,  Ford,  Slough,  &  Hurlock,  1972;  Hurlock  &  Slough,  1976),  evaluations  of 
training  for  particular  communities  (e.g.,  Kribs,  1976),  cost  effectiveness  of  CAI 
(Orlansky  &  String,  1979),  and  skills  ranging  from  procedural  (e.g.,  Fredericks  <5c  Hoover- 
Rice,  1977)  to  "hands-on"  performance  training  (e.g.,  Trollip,  1977).  These  efforts  have 
been  primarily  concerned  with  large,  time-sharing  systems  or  minicomputers. 

The  cost  and  training  effectiveness  of  these  experimental  efforts  has  been  questioned 
(Orlansky  &  String,  1979);  however,  this  kind  of  research  is  needed  to  improve  military 
training  as  well  as  to  explore  promising  technological  advances.  There  is  a  need,  at  this 
time,  to  evaluate  stand-alone  microprocessors  for  training  and  to  determine  the  most 
appropriate  training  materials  for  this  medium,  the  existence  of  any  maintenance  and  user 
problems,  and  software  development  costs  for  these  systems. 

One  of  the  few  training  efforts  involving  microcomputers  was  carried  out  at  this 
Center  by  Kelly,  Greitzer,  and  Hershman  (1981),  who  investigated  human  performance  in 
anti-air  warfare  threat  analysis.  An  interactive  air  defense  game  was  developed  and 
presented  on  the  computer  screen.  Research  subjects  played  the  game  while  the 
computer  presented  feedback  and  collected  data.  The  researchers  concluded  that 
proficient  performance  was  attained  and  that  the  game  provides  an  effective  vehicle  for 
human  performance  research.  The  results  demonstrated  the  flexibility  and  portability 
these  systems  offer  for  Navy  training. 

Purpose 

The  research  described  herein  is  part  of  a  3-year  developmental  effort.  The  overall 
purpose  is  to  evaluate  microprocessors  for  Navy  training  in  a  context  that  permits  the 


1 


exploration  of  training  techniques  related  to  AI  and  cognitive  science.  The  specific 
objectives  are:  (1)  to  create  a  usable  tactical  memorization  training  system,  (2)  to  gain 
experience  in  the  in  situ  development  of  stand-alone  CAI  systems  for  Navy  training,  (3)  to 
explore  gaming  techniques  to  facilitate  the  acquisition  of  declarative  information,  and  (4) 
to  use  this  developmental  situation  to  evolve  student  models  and  computer  tutors. 


APPROACH 


Training  Domain 

Tactical  action  officer  (TAO)  training,  which  requires  students  to  memorize  a  large 
body  of  facts  related  to  hostile  and  friendly  platforms  and  weapons  systems,  was  selected 
as  the  subject  matter  area  for  the  present  research  for  two  reasons:  (1)  it  is  a  domain  in 
which  combat  skills  are  crucial  and  (2)  it  appeared  that  results  obtained  might  be  useful 
for  both  initial  and  refresher  training.  The  TAO  is  a  multithreat  evaluator  and  decision 
maker  who  must  defend  his  ship  with  any  or  all  of  the  weapons  systems  or  platforms 
available  to  him.  He  must  have  a  thorough  knowledge  of  (1)  the  capabilities  and 
limitations  of  hostile  weapon  platforms  so  that  he  can  determine  the  existence  and 
priority  of  a  threat,  and  (2)  the  assets  of  his  own  ship  and  Navy  to  enable  him  to  counter 
that  threat  quickly. 

TAO  training  is  conducted  at  the  Fleet  Combat  Training  Centers,  Pacific  and 
Atlantic  (FCTCP  and  FCTCA),  and  at  the  Department  Head  Course  at  the  Surface 
Warfare  Officer  School.  Students  must  master  information  on  enemy  platforms,  weapons 
systems,  parameters  of  all  weapons  systems  (e.g.,  range,  speed,  flight  profile,  guidance 
systems,  etc.),  electronic  suites,  and  surveillance  capabilities.  They  must  master  the 
same  information  for  the  U.S.  Navy,  plus  theory  on  acoustic  and  electromagnetic 
propagation,  theory  on  missile  guidance  system  techniques,  and,  finally,  tactics  including 
such  topics  as  rules  of  engagement.  This  demanding  6-week  course  consists  of  lectures 
and  simulation  exercises. 

The  information  taught  to  TAOs  is  difficult  to  acquire  and  very  difficult  to  maintain 
unless  it  is  used  continually.  There  is,  therefore,  a  need  to  facilitate  the  initial 
acquisition  of  this  material,  as  well  as  to  provide  some  means  for  refresher  training:  A 
microprocessor-based  training  system  with  an  automated  tutor  might  be  a  good  solution 
for  both  needs.  Microcomputers  can  be  installed  easily  at  the  school,  in  ships,  or  at 
dockside.  Also,  the  automated  tutor  can  supervise  student  interactions  to  provide 
feedback  and  direction. 

Software  Development 


This  effort  involved  the  development  of  a  software  system  with  the  UCSD  PASCAL 
operating  system  on  a  small,  stand-alone  computer  system.  The  software,  which  was 
designed  to  provide  drill  and  practice  through  game  playing  with  a  data  base  of  Soviet 
Navy  platforms  and  weapons,  has  been  implemented  at  FCTCP  as  part  of  the  TAO 
training  course. 

The  TERAK  microcomputer,1  an  LSI-ll-based  dual  floppy  disk  drive  system  with  32k 
word  memory  capability,  was  selected  for  the  initial  phase  of  this  research.  It  includes  a 


identification  of  the  equipment  is  for  documentation  purposes  only.  It  does  not 
constitute  endorsement  by  the  Navy  Department,  since  many  similar  microcomputers 
could  serve  the  same  purpose. 
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CRT  display  for  the  presentation  of  graphics  and  text  in  black  and  white.  Eventually,  the 
software  implemented  on  the  TERAK  will  be  transferred  to  the  PERQ  system,  which  has 
128k  words  of  memory,  a  24-megabyte  Winchester  disk,  and  high  resolution  graphics 
(1024x768).  The  PERQ  will  provide  the  memory  capacity  required  for  the  development  of 
student  models  and  a  more  advanced  automated  tutor. 


Data  Base 


A  data  base  was  developed  of  all  Soviet  threat  information  taught  in  the  TAO  course. 
This  consists  of  a  series  of  facts  about  Soviet  platforms,  weapons  systems,  and  the 
parameters  associated  with  both.  Since  the  data  base  was  compiled  and  is  being 
maintained  by  instructors  from  the  TAO  school  at  FCTCP,  it  reflects  the  information 
being  taught  at  the  school  at  any  given  time. 

The  data  base  was  structured  as  a  semantic  network  and  then  entered  into  the 
system.  The  data  in  a  semantic  network  consist  of  nodes  that  represent  objects  and  links 
between  those  nodes  that  show  relationships  between  the  objects.  For  example,  in  the 
data  base  for  this  project,  the  two  objects  "ship"  and  "radar"  are  related  by  "has,"  allowing 
the  system  to  record  the  fact  that  ships  have  radars.  Beginning  with  the  definition  of  all 
of  the  objects  in  the  data  base,  a  hierarachy  is  developed  in  which  objects  and  their 
relationships  are  specifically  defined.  Figure  1  shows  the  definitions  associated  with  the 
fact  that  "KYNDA  has  peel  group."  Figure  2  shows  all  of  the  information  recorded  about 
the  KYNDA.  A  typical  data  base  contains  representations  of  several  thousand  such  facts. 
Not  all  of  the  entries  will  be  defined  as  objects.  Some  of  them  are  attributes,  such  as 
"KYNDA  speed  30  kts,"  where  KYNDA  is  an  object,  speed  is  a  relationship,  and  30  kts  is 
an  attribute. 
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Figure  1.  Sample  semantic  network  triangle 
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Figure  2.  Sample  semantic  network  node. 


The  data  base  also  includes  synonyms  and  quantifiers.  Synonyms  allow  the  same 
object  to  be  referred  to  in  multiple  ways.  A  limited  form  of  quantification  is 
implemented  so  that  the  number  of  radars  or  other  objects  associated  with  some 
particular  object  can  be  represented. 

Computer  Games 


Given  this  kind  of  representation  of  the  data  base,  it  is  then  possible  for  additional, 
independent  software,  or  interpretive  procedures,  to  interact  with  and  manipulate  the 
data  base  in  an  intelligent  fashion.  For  example,  the  interpretive  software  can  make 
inferences  about  the  data  base;  it  "knows"  enough  about  the  structure  of  the  data  to  ask 
questions  such  as  "What  radars  does  the  KYNDA  carry?"  The  software  in  this  system 
consists  of  a  series  of  games  that  quiz  the  student  about  the  data  base.  Since  these  games 
are  separate  from  the  data  base,  they  will  run  on  any  data  base  that  can  be  represented  in 
the  semantic  network  format  just  described.  In  quizzing  the  student,  the  gaming  software 
draws  items  randomly  from  the  data  base  and  generates  questions,  examples,  or  graphics. 
Unlike  previous  computer-based  training,  these  exercises  are  not  specifically  prepro¬ 
grammed. 

The  following  games  are  available: 

1.  Flash.  The  program  quizzes  the  player  about  facts  in  the  data  base.  Students 
may  choose  to  have  the  questions  presented  in  either  the  recall  or  recognition  mode. 
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2.  Twenty  questions.  The  computer  "thinks"  of  an  item  and  the  player  must  guess 
what  it  is  by  partitioning  the  data  base.  The  student  might  ask,  "Is  the  item  carried  on 
cruisers?"  The  computer  will  then  respond  with  a  true  or  false  answer  and  the  student  can 
proceed  with  more  questions. 

3.  Concentration.  The  student  is  presented  with  a  list  of  items.  He  must  match 
those  items  that  are  related  by  common  characteristics. 

Constraints.  The  student  is  required  to  narrow  down  a  list  of  objects  by  knowing 
what  they  have  in  common.  This  is  very  much  like  the  actual  TAO  task  in  that  the  TAO 
typically  continues  to  receive  bits  of  information  until  he  can  identify  the  threat.  For 
example,  if  a  certain  type  of  radar  emission  is  received,  the  TAO  must  know  which 
platforms  in  his  area  could  be  sending  that  emission.  Usually,  reception  of  several  more 
emissions  are  necessary  to  identify  the  specific  threat. 

5.  Matrix.  The  student  must  fill  in  the  answers  to  threat  matrices  chosen  either  by 
the  instructor  or  himself.  The  matrix  has  column  and  row  headings  chosen  from  the  data 
base  (e.g.,  KYNDA  and  radar)  and  the  student  must  fill  in  the  boxes  (e.g.,  peel  group). 

6.  Show.  The  program  presents  a  graphic  representation  of  the  data  base.  It  is 
more  of  a  reference  function  than  a  game.  When  the  student  enters  "KYNDA,"  the 
system  displays  everthing  it  knows  about  that  ship,  as  in  Figure  2. 

7.  Picture  quiz.  This  game  presents  graphic  representations  of  ships  in  the  data 
base.  The  student  is  supposed  to  identify  the  ships  or  their  individual  components. 

The  interface  to  the  game  system  has  been  the  focus  of  much  effort  in  this  project. 
The  keyboard  with  the  TERAK  system  has  the  standard  alphanumeric  character  keys,  plus 
several  others  dedicated  to  special  functions  in  an  attempt  to  facilitate  the  use  of  the 
interface.  The  student  uses  these  special  keys  whenever  possible,  making  typewritten 
entries  only  where  necessary.  There  are  horizontal-  and  vertical-arrow  keys  that  move 
the  cursor  around  the  screen  and  that  are  used  primarily  for  selecting  responses.  There 
are  also  "tell  me,"  "hints,"  "quit,"  "select,"  and  "help"  keys,  which  are  functional  in  all  of 
the  games. 

Extensive  use  has  been  made  of  menus,  or  lists  of  options,  for  the  system  interface. 
The  student  selects  from  menus  the  game  he  wants  to  play,  the  category  of  the  data  base 
he  wants  to  be  quizzed  on  (e.g.,  missiles,  any  subcategories  such  as  surface-to-surface 
missiles)  and,  in  many  cases,  possible  answers  to  questions.  To  use  these  menus,  the 
student  simply  positions  the  cursor  next  to  the  desired  response  and  presses  "select." 

Procedure 


Audit  trails  were  developed  to  collect  information  on  all  student  interactions  with 
the  system.  They  recorded  (1)  the  games  played,  (2)  the  data  base  categories  selected,  (3) 
the  questions,  hints,  help,  and  answers  generated  by  the  system,  (4)  students'  answers  and 
options  exercised  to  quit,  and  (5)  time  spent  on  the  system. 

All  developmental  work  was  carried  out  in  situ;  that  is,  in  the  school  setting.  There 
was  extensive  participation  by  FCTCP  students  and  instructors  from  the  beginning  of  the 
project.  Ideas  were  discussed  with  the  users  before  work  was  started;  they  compiled  the 
data  base;  and,  early  in  development,  one  TERAK  was  installed  at  the  school.  Because 
informal  feedback  was  solicited  throughout  the  development,  changes  were  continually 
being  made  to  the  system. 
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Once  the  games  were  in  reasonable  working  order,  three  more  TERAKs  were  installed 
at  the  school  for  the  students'  use.  The  students  participate  in  simulation  exercises  and 
receive  lectures  every  day.  At  the  end  of  the  day,  they  study  before  going  home.  The 
TERAKs  were  available  for  those  who  wanted  to  use  them  to  study,  but  no  one  was 
required  to  use  them  if  they  preferred  other  study  methods.  Students  who  used  the 
systems  were  told  that  the  development  was  still  in  the  early  stages  and  their  critical 
comments  were  encouraged.  The  games  were  available  to  three  classes  of  students 
(N  =  54). 

Audit  trail  data,  school  test  scores,  and  attitude  questionnaire  responses  were 
collected  from  all  students.  These  data  will  be  used  in  the  future  for  the  development  of 
student  models  and  the  automated  tutor,  as  well  as  for  a  formal  evaluation  of  training 
effectiveness.  The  questionnaire  data,  in  combination  with  comments  solicited  from 
students  and  instructors  throughout  the  development,  were  used  for  formative  evaluation. 

Other  data  bases  were  also  compiled  and  implemented  to  test  the  generalizability  of 
the  gaming  system.  These  data  bases  contained  facts  related  to  South  American 
geography  and  electronic  warfare  threat  evaluation. 

At  this  time,  formative  evaluation  is  continuing.  Much  of  the  primary  efforts  are 
directed  toward  converting  the  software  to  run  on  the  PERQ  system. 


FINDINGS  AND  DISCUSSION 


In  Situ  Development 

In  situ  development  of  the  tactical  training  system  created  an  environment  in  which 
continuous  and  extensive  input  from  users  was  provided  to  the  system  designers.  From 
the  conceptualization  stages  through  implementation  of  computer-based  materials  at 
FCTCP,  students  and  instructors  were  involved  in  each  phase  of  the  work.  This  differs 
markedly  from  the  approach  typically  taken  in  developing  CAI. 

The  iterative  changes  possible  with  in  situ  development  seemed  to  be  a  better,  more 
efficient  approach  for  producing  the  end  product.  This  form  of  development  encouraged 
close  interactions  with  potential  system  users  during  the  critical  early  design  stages  and 
allowed  the  interaction  to  continue  throughout  the  project.  As  a  result,  the  end  product 
reflects  user  needs,  rather  than  researcher  ideas  of  user  needs.  Also,  since  the  user 
community  participated  in  shaping  the  materials,  there  were  involvement  and  support  at 
the  time  of  implementation  for  student  testing. 

The  feedback  received  on  the  gaming  system  throughout  its  development  ranged  from 
very  informal  feedback  from  both  students  and  instructors  to  responses  on  the  attitude 
questionnaire  administered  to  the  three  classes  of  students.  This  feedback  resulted  in 
major  changes  in  the  games,  the  game  instructions,  and,  primarily,  the  system  interface. 
These  changes  will  be  described  in  the  following  section. 

Tactical  Gaming  System 

The  TAO  course  is  conducted  in  a  very  high  pressure  environment  since  a  massive 
amount  of  information  must  be  mastered  in  6  weeks.  As  a  result,  students  are  not  willing 
to  spend  much  time  learning  anything  more  than  is  required  for  the  class.  Thus,  in  the 
early  stages  of  development,  not  very  many  students  learned  to  use  the  computer-based 
system  as  a  study  aid.  Rather,  they  would  return  to  the  reliable  study  methods  (i.e., 
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practicing  with  flash  cards,  devising  threat  matrices,  and  reviewing  class  notes)  they  had 
used  before. 

If  the  games  were  hard  to  understand,  students  ignored  them.  The  students  did  not 
like  a  game  called  "browse,"  even  after  the  interface  had  been  simplified.  The 
researchers  had  felt  that  this  would  be  a  very  well-liked  game  because  it  appeared  to  be  a 
valuable  study  tool.  Apparently,  however,  students  felt  too  much  explanation  was 
required  before  they  could  understand  how  to  use  the  game.  This  was  a  normal  reaction, 
because  the  students  had  not  been  presented  with  material  that  convinced  them  that  the 
computer-based  system  was  a  superior  study  aid.  They  felt  that,  even  if  they  were  very 
familiar  with  how  to  use  CAI,  it  was  no  better  than  the  study  methods  they  were  using. 
The  people  who  did  use  the  gaming  system  had  much  different  feelings  about  it;  their 
attitudes  will  be  described  later. 

The  complaints  made  by  students  who  used  the  system  motivated  many  changes  in  the 
system  interface.  One  very  common  complaint  was  that  too  many  manipulations  were 
required  before  one  could  begin  studying.  In  the  early  stages  of  the  devleopment, 
considerable  use  of  the  keyboard  was  required  to  move  around  the  system.  This  was 
changed  by  (1)  implementing  menus  wherever  possible,  and  (2)  designating  special  key 
functions  (by  labeling  existing  keys).  Except  for  answers  required  for  some  of  the 
questions,  the  student  is  required  to  do  very  little  typing.  Instead,  he  positions  the  cursor 
next  to  the  desired  option  and  presses  the  "select"  button.  In  most  cases,  the  student  only 
has  to  select  the  game,  the  data  base,  and  the  categories  before  beginning  to  play  the 
game. 

Another  common  complaint  was  that  the  system  was  not  very  flexible  in  terms  of  the 
input  it  would  accept.  To  eliminate  this  problem,  a  spelling  corrector  was  developed  and 
an  extensive  list  of  synonyms  was  put  in  the  data  base. 

The  items  in  this  particular  data  base  are  subject  to  change  and,  although  there  were 
never  more  than  one  or  two  inaccuracies,  students  were  very  intolerant  if  there  were  any. 
As  a  result,  a  data  base  editor  was  developed  for  use  by  school  personnel  to  update  and 
change  the  data  base  rapidly. 

The  students  did  not  appear  to  be  very  interested  in  anything  not  strictly  designed  to 
facilitate  memorization  of  the  facts  needed  for  the  school  tests.  The  constraints  game, 
which  was  most  like  a  TAO  task,  was  not  rated  as  highly  as  were  the  other  games.  The 
idea  for  this  game  had  been  suggested  by  a  fleet  TAO,  who  believed  that  this  kind  of 
development  was  really  needed  and  would  be  highly  motivating  to  keep  requisite  skills  up 
to  date.  It  may  be  that  this  kind  of  game  would  be  more  suitable  for  those  who  have 
already  been  through  the  course  to  use  aboard  ship  or  in  other  refresher  applications. 
Similarly,  from  talking  to  instructors  and  fleet  TAOs,  it  was  assumed  that  scoring  the 
games  would  be  a  motivating  feature  because  it  would  introduce  a  competitive  element. 
However,  the  students  felt  quite  differently;  the  majority  felt  that  scoring  was  not 
necessary. 

Ideas  for  new  games  were  additional  results  of  performing  this  work  in  situ.  The 
picture  quiz  game,  for  example,  resulted  from  one  instructor's  opinion  that  the  ability  to 
recognize  a  ship  from  its  silhouette  is  related  to  a  good  working  knowledge  of  the  data 
base. 

The  instructions  also  had  to  be  changed  to  suit  student  needs.  Originally,  a  lengthy 
hard  copy  of  system  instructions,  including  instructions  and  objectives  for  each  game,  was 
placed  beside  each  terminal.  3ust  as  with  the  interface,  students  did  not  want  to  be 
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bothered  with  anything  they  did  not  need  to  know.  As  a  result,  the  instructions  were 
shortened  considerably  and  put  on-line.  Only  the  essential  information  was  presented  and 
then  only  at  the  time  and  location  when  needed.  It  was  presented  either  as  a  prompt 
always  on  the  screen  or  as  a  short  sentence  accessed  by  the  "help"  prompt. 

The  students  who  chose  to  use  the  system  tended  to  be  those  who  scored  highest  on 
the  class  tests  and/or  had  some  experience  with  computers  before  the  TAO  class.  In  any 
case,  they  were  clearly  comfortable  taking  the  time  to  learn  to  use  the  system.  Once 
these  students  were  working  on  the  system,  they  tended  to  continue  using  it  several  hours 
a  day  after  class  lectures.  They  said  that  they  liked  the  system  for  the  following  kinds  of 
reasons: 

•  It  is  good  for  asking  questions  you  would  not  ask  yourself. 

•  It  is  faster  than  self-study. 

•  You  can  cover  more  information  this  way. 

•  It  points  out  your  weak  areas. 

These  students  were  asked  to  rate  the  games  on  a  scale  that  included  the  following  points: 
very  effective,  effective,  somewhat  effective,  ineffective,  very  ineffective,  and  did  not 
play.  Most  of  the  students  rated  the  games  "effective." 

Although  the  data  collected  by  the  audit  trails  have  not  yet  been  evaluated,  the 
implementation  of  audit  trails  for  student  runs  has  provided  some  important  information. 
For  example,  an  excessive  amount  of  data  were  specified  in  terms  of  what  could 
realistically  be  used.  This  aspect  of  the  data  collection  effort  had  no  effect  on  students. 
While  they  were  aware  that  the  system  was  keeping  records  of  their  inputs,  the  process 
was  not  intrusive.  Data  collected  in  this  manner  will  eventually  be  used  for  developing 
student  models  and  as  input  for  the  evaluation  of  training  effectiveness. 

Microprocessors  for  Navy  Training 

The  microprocessor  system  used  in  this  effort  was  very  effective  for  the  work  done 
to  this  point  in  time.  The  physical  configuration  of  such  desk-top  systems  makes  them 
highly  portable.  As  they  are  small  and  need  only  a  few  plugs  for  connections,  they  are 
very  easy  to  move  for  repair  or  relocation. 

Given  the  eventual  goal  of  developing  student  models  and  a  sophisticated  automated 
tutor,  the  TERAK  system  has  reached  the  limits  of  its  memory  capabilities  for  this 
project.  It  did,  however,  prove  very  capable  of  handling  a  variety  of  features  directly 
related  to  user  acceptance  (e.g.,  the  data  base  editor,  the  spelling  corrector,  etc.).  These 
features  were  additions  to  what  was  already  a  very  unique  and  advantageous  approach  to 
computer-based  training;  that  is,  using  a  semantic  network  to  represent  the  data  base. 
Thus,  a  small  stand-alone  microprocessor  system  has  been  shown  to  be  capable  of 
providing  fairly  sophisticated  computer-based  training  that  is  considered  useful  by  the 
Navy  users. 

Semantic  Networks 


The  major  advantages  of  using  semantic  networks  for  computer-based  instruction  are 
that  (1)  the  degree  of  individualization  provided  is  greater  than  that  which  the  traditional 
approach  would  provide  in  proportion  to  the  programming  required,  and  (2)  the  interpre¬ 
tive  procedures  developed  can  be  generalized  to  any  other  data  base  that  can  be 
represented  as  a  semantic  network. 
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The  semantic  network  structures  the  data  base  so  that  the  gaming  software  can  draw 
items  from  the  data  base  as  needed.  This  permits  students  to  progress  through  drill  and 
practice  on  the  declarative  information  whenever  and  as  often  as  they  wish,  and  lets  them 
omit  any  questions  or  sequences  in  which  they  are  not  interested.  The  standard  approach 

to  development  of  computer-based  instruction  does  not  easily  provide  this  kind  of 
flexibility. 

The  implementation  of  additional  data  bases  performed  as  part  of  this  effort 
demonstrated  the  generalizability  of  the  gaming  software.  Once  the  data  bases  were 
compiled,  implementation  consisted  of  typing  the  entries  into  the  system.  The  result  was 
that,  with  the  exception  of  games  using  graphics,  a  new  series  of  games  was  easily  and 
quickly  created  for  each  new  subject  area. 


CONCLUSIONS 

The  research  described  in  this  report  focused  on  the  combined  elements  of  in  situ 
development  of  computer-based  training,  gaming  for  memorization  of  declarative  infor¬ 
mation,  methodologies  related  to  intelligent  computer-assisted  instruction,  microcom¬ 
puter  systems  for  training,  and  development  of  computer-based  drill  and  practice 
materials  to  supplement  (rather  than  replace)  traditional  forms  of  Navy  training. 
Although  the  evaluation  of  the  impact  of  these  elements  on  training  effectiveness  has  not 
yet  been  completed,  some  comments  can  be  made  about  what  has  been  learned  over  the 
past  2 Vi  years  of  development: 

1.  The  system  that  was  designed  is  useful  for  memorizing  declarative  information. 
Once  students  start  using  the  system,  they  continue  to  use  it  and  report  that  it  is  an 
effective  study  aid. 

2.  Gaming  seems  to  be  a  reasonable  approach  for  drill  and  practice  of  this  kind  of 

information.  To  evaluate  the  effect  of  games  fully  will  require  the  collection  of  some 
specific  information  about  what  makes  the  games  interesting  for  this  particular  group  of 
students.  Early  information,  for  example,  suggested  that  competition  is  not  a  necessary 
element.  J 


3.  The  use  of  the  semantic  network  permitted  increased  individualization  in  that  it 
provided  enough  flexibility  to  serve  the  wide  range  of  abilities  brought  to  training. 
Additionally,  once  the  software  was  developed  for  the  semantic  network  and  the  games  it 
was  readily  generalized  to  other  data  bases. 

4.  Microprocessors  offer  several  advantages  over  larger  systems  for  Navy  training. 
They  are  portable  and  easily  repaired.  Additionally,  the  users  were  able  to  take  full 
responsibility  for  running  students  and  maintaining  the  data  base. 

5.  In  situ  development  was  convenient  for  the  researchers  and  had  several  advan- 
tages  for  the  users.  The  resultant  end  product  is  one  that  reflects  user  needs  and, 
therefore,  has  their  support. 


FUTURE  DIRECTION 

The  gaming  system  is  being  transitioned  into  advanced  development  funding  as  part 
of  the  Computer-based  Educational  Support  System  project. 
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